Advanced CI

Part 2, Chapter 16


In this chapter, we'll update the CI process and set up a multistage Docker build for production.


Multistage Docker Build

Start by updating Dockerfile.prod like so:

###########
# BUILDER #
###########

# pull official base image
FROM python:3.10.1-slim-buster as builder

# install system dependencies
RUN apt-get update \
  && apt-get -y install gcc postgresql \
  && apt-get clean

# set work directory
WORKDIR /usr/src/app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt

# lint
COPY . /usr/src/app/
RUN pip install black==21.12b0 flake8==4.0.1 isort==5.10.1
RUN flake8 .
RUN black --exclude=migrations .
RUN isort .


#########
# FINAL #
#########

# pull official base image
FROM python:3.10.1-slim-buster

# create directory for the app user
RUN mkdir -p /home/app

# create the app user
RUN addgroup --system app && adduser --system --group app

# create the appropriate directories
ENV HOME=/home/app
ENV APP_HOME=/home/app/web
RUN mkdir $APP_HOME
WORKDIR $APP_HOME

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV ENVIRONMENT prod
ENV TESTING 0

# install system dependencies
RUN apt-get update \
  && apt-get -y install netcat gcc postgresql \
  && apt-get clean

# install python dependencies
COPY --from=builder /usr/src/app/wheels /wheels
COPY --from=builder /usr/src/app/requirements.txt .
RUN pip install --upgrade pip
RUN pip install --no-cache /wheels/*
RUN pip install "uvicorn[standard]==0.16.0"

# add app
COPY . .

# chown all the files to the app user
RUN chown -R app:app $HOME

# change to the app user
USER app

# run gunicorn
CMD gunicorn --bind 0.0.0.0:$PORT app.main:app -k uvicorn.workers.UvicornWorker

Here, we used a Docker multistage build to reduce the final image size. Essentially, builder is a temporary image that's used for building the Python wheels. The wheels are then copied over to the final production image and the builder image is discarded.

Did you notice that Flake8, Black, and isort are being run in the builder image? If any of them fail, the build stops.

To test locally, build the new image and spin up the container:

$ docker build -f project/Dockerfile.prod -t web ./project

$ docker run --name fastapi-tdd -e PORT=8765 -e DATABASE_URL=sqlite://sqlite.db -p 5003:8765 web:latest

You should see something similar to:

[2021-12-28 00:30:21 +0000] [8] [INFO] Starting gunicorn 20.1.0
[2021-12-28 00:30:21 +0000] [8] [INFO] Listening at: http://0.0.0.0:8765 (8)
[2021-12-28 00:30:21 +0000] [8] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-12-28 00:30:21 +0000] [9] [INFO] Booting worker with pid: 9
[2021-12-28 00:30:22 +0000] [9] [INFO] Started server process [9]
[2021-12-28 00:30:22 +0000] [9] [INFO] Waiting for application startup.
[2021-12-28 00:30:22 +0000] [9] [INFO] Application startup complete.

Navigate to http://localhost:5003/ping/.

You should see:

{
  "ping": "pong!",
  "environment": "prod",
  "testing": false
}

Bring down the container once done:

$ docker rm fastapi-tdd -f

Commit and push your code up to GitHub to trigger a new build. Ensure it passes.

Docker Caching

Next, to speed up the build on GitHub Actions, update the .github/workflows/main.yml file:

name: Continuous Integration and Delivery

on: [push]

env:
  IMAGE: docker.pkg.github.com/$(echo $GITHUB_REPOSITORY | tr '[A-Z]' '[a-z]')/summarizer

jobs:

  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2.4.0
        with:
          ref: main
      - name: Log in to GitHub Packages
        run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - name: Pull images
        run: |
          docker pull ${{ env.IMAGE }}-builder:latest || true
          docker pull ${{ env.IMAGE }}-final:latest || true
      - name: Build images
        run: |
          docker build \
            --target builder \
            --cache-from ${{ env.IMAGE }}-builder:latest \
            --tag ${{ env.IMAGE }}-builder:latest \
            --file ./project/Dockerfile.prod \
            "./project"
          docker build \
            --cache-from ${{ env.IMAGE }}-final:latest \
            --tag ${{ env.IMAGE }}-final:latest \
            --file ./project/Dockerfile.prod \
            "./project"
      - name: Push images
        run: |
          docker push ${{ env.IMAGE }}-builder:latest
          docker push ${{ env.IMAGE }}-final:latest

  test:
    name: Test Docker Image
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Checkout
        uses: actions/checkout@v2.4.0
        with:
          ref: main
      - name: Log in to GitHub Packages
        run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - name: Pull images
        run: |
          docker pull ${{ env.IMAGE }}-builder:latest || true
          docker pull ${{ env.IMAGE }}-final:latest || true
      - name: Build images
        run: |
          docker build \
            --target builder \
            --cache-from ${{ env.IMAGE }}-builder:latest \
            --tag ${{ env.IMAGE }}-builder:latest \
            --file ./project/Dockerfile.prod \
            "./project"
          docker build \
            --cache-from ${{ env.IMAGE }}-final:latest \
            --tag ${{ env.IMAGE }}-final:latest \
            --file ./project/Dockerfile.prod \
            "./project"
      - name: Run container
        run: |
          docker run \
            -d \
            --name fastapi-tdd \
            -e PORT=8765 \
            -e ENVIRONMENT=dev \
            -e DATABASE_URL=sqlite://sqlite.db \
            -e DATABASE_TEST_URL=sqlite://sqlite.db \
            -p 5003:8765 \
            ${{ env.IMAGE }}-final:latest
      - name: Pytest
        run: docker exec fastapi-tdd python -m pytest .
      - name: Flake8
        run: docker exec fastapi-tdd python -m flake8 .
      - name: Black
        run: docker exec fastapi-tdd python -m black . --check
      - name: isort
        run: docker exec fastapi-tdd python -m isort . --check-only

  deploy:
    name: Deploy to Heroku
    runs-on: ubuntu-latest
    needs: [build, test]
    env:
      HEROKU_APP_NAME: sleepy-thicket-77414
      HEROKU_REGISTRY_IMAGE: registry.heroku.com/${HEROKU_APP_NAME}/summarizer
    steps:
      - name: Checkout
        uses: actions/checkout@v2.4.0
        with:
          ref: main
      - name: Log in to GitHub Packages
        run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - name: Pull images
        run: |
          docker pull ${{ env.IMAGE }}-builder:latest || true
          docker pull ${{ env.IMAGE }}-final:latest || true
      - name: Build images
        run: |
          docker build \
            --target builder \
            --cache-from ${{ env.IMAGE }}-builder:latest \
            --tag ${{ env.IMAGE }}-builder:latest \
            --file ./project/Dockerfile.prod \
            "./project"
          docker build \
            --cache-from ${{ env.IMAGE }}-final:latest \
            --tag ${{ env.IMAGE }}:latest \
            --tag ${{ env.HEROKU_REGISTRY_IMAGE }}:latest \
            --file ./project/Dockerfile.prod \
            "./project"
      - name: Log in to the Heroku Container Registry
        run: docker login -u _ -p ${HEROKU_AUTH_TOKEN} registry.heroku.com
        env:
          HEROKU_AUTH_TOKEN: ${{ secrets.HEROKU_AUTH_TOKEN }}
      - name: Push to the registry
        run: docker push ${{ env.HEROKU_REGISTRY_IMAGE }}:latest
      - name: Set environment variables
        run: |
          echo "HEROKU_REGISTRY_IMAGE=${{ env.HEROKU_REGISTRY_IMAGE }}" >> $GITHUB_ENV
          echo "HEROKU_AUTH_TOKEN=${{ secrets.HEROKU_AUTH_TOKEN }}" >> $GITHUB_ENV
      - name: Release
        run: |
          chmod +x ./release.sh
          ./release.sh

We're now pulling, building, and pushing the builder image using the --target option.

For more on Docker caching, review the Faster CI Builds with Docker Cache article.

Update Requirements

Next, let's move development-only requirements to a new file.

requirements-dev.txt:

black==21.12b0
flake8==4.0.1
isort==5.10.1
pytest==6.2.5
pytest-cov==3.0.0
pytest-xdist==2.5.0

-r requirements.txt

requirements.txt:

aerich==0.6.1
asyncpg==0.25.0
fastapi==0.70.1
gunicorn==20.1.0
newspaper3k==0.2.8
requests==2.26.0
tortoise-orm==0.17.8
uvicorn==0.16.0

Update Dockerfile:

# pull official base image
FROM python:3.10.1-slim-buster

# set working directory
WORKDIR /usr/src/app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# install system dependencies
RUN apt-get update \
  && apt-get -y install netcat gcc postgresql \
  && apt-get clean

# install python dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
COPY ./requirements-dev.txt .
RUN pip install -r requirements-dev.txt

# add app
COPY . .

# add entrypoint.sh
COPY ./entrypoint.sh .
RUN chmod +x /usr/src/app/entrypoint.sh

# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

Test:

$ docker-compose down -v
$ docker-compose up -d --build
$ docker-compose exec web aerich upgrade

$ docker-compose exec web python -m pytest
$ docker-compose exec web flake8 .
$ docker-compose exec web black .
$ docker-compose exec web isort .

Install the requirements in the test job of .github/workflows/main.yml:

- name: Install requirements
  run: docker exec fastapi-tdd pip install black==21.12b0 flake8==4.0.1 isort==5.10.1 pytest==6.2.5

Full job:

test:
  name: Test Docker Image
  runs-on: ubuntu-latest
  needs: build
  steps:
    - name: Checkout
      uses: actions/checkout@v2.4.0
      with:
        ref: main
    - name: Log in to GitHub Packages
      run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    - name: Pull images
      run: |
        docker pull ${{ env.IMAGE }}-builder:latest || true
        docker pull ${{ env.IMAGE }}-final:latest || true
    - name: Build images
      run: |
        docker build \
          --target builder \
          --cache-from ${{ env.IMAGE }}-builder:latest \
          --tag ${{ env.IMAGE }}-builder:latest \
          --file ./project/Dockerfile.prod \
          "./project"
        docker build \
          --cache-from ${{ env.IMAGE }}-final:latest \
          --tag ${{ env.IMAGE }}-final:latest \
          --file ./project/Dockerfile.prod \
          "./project"
    - name: Run container
      run: |
        docker run \
          -d \
          --name fastapi-tdd \
          -e PORT=8765 \
          -e ENVIRONMENT=dev \
          -e DATABASE_URL=sqlite://sqlite.db \
          -e DATABASE_TEST_URL=sqlite://sqlite.db \
          -p 5003:8765 \
          ${{ env.IMAGE }}-final:latest
    - name: Install requirements
      run: docker exec fastapi-tdd pip install black==21.12b0 flake8==4.0.1 isort==5.10.1 pytest==6.2.5
    - name: Pytest
      run: docker exec fastapi-tdd python -m pytest .
    - name: Flake8
      run: docker exec fastapi-tdd python -m flake8 .
    - name: Black
      run: docker exec fastapi-tdd python -m black . --check
    - name: isort
      run: docker exec fastapi-tdd python -m isort . --check-only

Again, commit and push your code up to GitHub to trigger a new build. Make sure it passes before moving on.




✓ Mark as Completed